WLD-Reg: A Data-Dependent Within-Layer Diversity Regularizer

Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with gradient-based optimization, where the errors back-propagated from last layer back to first one. At each optimization step, neurons at given receive feedback belonging higher hierarchy. In this paper, we propose complement traditional 'between-layer' additional 'within-layer' encourage diversity activations within same layer. To end, measure pairwise similarity between outputs and use it model layer's overall diversity. We present an extensive empirical study confirming that proposed approach enhances performance several state-of-the-art neural network models tasks. The code is publically available https://github.com/firasl/AAAI-23-WLD-Reg.